Voice Orchestrated Infrastructure System

ABSTRACT

There is provided a voice orchestrated infrastructure system which includes a hub in communication with at least one endpoint device located in a room or area, and the at least one endpoint device is in communication with the hub and at least one endpoint device in a second room or area through the hub. The hub includes a set of non-transitory commands which when executed with a central processor the at least one endpoint device is activated and controlled by voice commands which are independent of service provider type. The hub includes a non-transitory computer-readable storage medium which stores computer-executable instructions that, when executed by a processor, cause the processor to perform operations for determining the voice command which is communicated to and from the at least one end point device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims benefit of and priority to U.S. application Ser.No. 16/775,240 filed on Jan. 28, 2020, which claims benefit of andpriority to U.S. Provisional Application No. 62/797,949 filed on Jan.29, 2019, each of which are incorporated herein by reference in theirentirety.

BACKGROUND

Smart home devices provide convenience and productivity for users, butcompatibility of different brands arise for each endpoint device. Thereis a need for smart home control which is independent of the personalassistant device that is used with the endpoint devices.

SUMMARY OF THE INVENTION

There is provided a voice orchestrated infrastructure system for usewith and in creating smart homes that are controlled by one or moreauthorized users from a centralized hub device. For one or more of therooms in a residence or dwelling, each of the rooms has embedded orfastened in fixtures and devices within the room, microphones andspeakers which are in communication with the central hub system and alsowith each other through the central hub system via wi-fi networking. Thesystem of the present invention is not dependent on any particular brandof voice controlled personal assistant device (such as Siri/Alexa/Nest).Microphones/speakers/ video are all controlled and communicated directlythrough one hub. Service provider that is utilized does not matter. Asthe voice orchestrated infrastructure is agnostic as to the system ortype of personal assistant device employed by the user(s).

The system has Wi-Fi capability to talk to the hub and authorizeddevices. Motion detection via sound effects to activate the roomdevices. All privacy is controlled through the hub, along with securityfeatures. Communication system protocol—devices in each room of house ordwelling acting as a telephone.

Voice command is directed to an appropriate destination, such as a room,or particular endpoint device in a room. This includes lights,thermostats, electric outlets, appliances—washer, dryer, stove,refrigerator, oven, range, automated vacuums. Security systems forwindows and doors, motion detectors, smoke detectors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a hub connected to one or more rooms eachwith endpoint devices;

FIG. 2 is a schematic of the voice orchestrated infrastructure bridgecomponents.

FIG. 3 is a diagram of the bridge components showing drivers, logiclayers, and network layers;

FIG. 4 is a diagram of the bridge system components;

FIG. 5 is a schematic of the computer device components of the presentinvention.

DETAILED DESCRIPTION

As illustrated in FIG. 1 for the present invention 10, each of the Roomor area 1 (14), Room/area 2 (16), and Room/area 3 (18) and a pluralityof other rooms or areas, designated as room or area N (20), areconnected and in communication to the hub 12, with each room or areahaving one or more endpoint devices (EPD) 22, 24, 26, and 28, such alight switches, outlets, appliances etc. All endpoints 22, 24, 26, and28 are voice orchestrated or controlled and have microphones andspeakers at the endpoints 22, 24, 26, and 28 for communication with,from and back to the hub 12. Through the hub 12, communication can bemade to and from any room 14, 16, 18 or 20 for activating ordeactivating or adjusting/controlling any device or endpoint 22, 24, 26,and 28 in the room. The system 10 can be synched and controlled withlaptop or hand held devices as well whether by voice control orapplications. Proprietary software and rules are designed for the huband system to execute the system of the present invention.

Bridge Description:

Referring to FIG. 2, there is shown the VOI bridge components 32. TheVOI bridge is a small-sized device based on Espressif ESP-32 chip(eXtensa ESP32) 36. The bridge 32 consisting an array of MEMSmicrophones 42 connected to an audio codec 34 and an ESP32 Wi-Fi/BTenabled 32 bit microcontroller. The MEMs microphone array on the bridgeallows you to leverage voice recognition in your app creations by usingthe latest online cognitive services including Microsoft CognitiveService, Amazon Alexa Voice Service, Google Speech API, Wit.ai andHoundify. The bridge provides to users the opportunity to integratecustom voice and hardware-accelerated machine learning technology rightonto the silicon. It's for makers, industrial and home IoT engineers. Itallows for triggering events based on sound detections, such asreceiving a text message when your dog is barking back home. One of theexamples of working with bridge—you can build your own Amazon's Alexausing the Bridge 32. Bridge contains the following peripherals: ac/dcpower converter 38, 46; general purpose input/output 52, universalasynchronous receiver transmitter (UART) 50, analog-digital converter(ADC) 54, voice/sound streaming information 42, 44, 48; networkinterface; status indicators; control buttons; low power drivers forcontrol external devices 40 (optional); may have wireless 56 interfaceson-board such as Bluetooth/ZigBee/Z-Wave (optional). External audiocodec 34 is used for input/output 42, 44 and coding/decoding ofvoice/sound information 48. Bridge can work/have internal and externalmicrophones and built-in speaker.

In an embodiment, the end points 22, 24, 26, 28 include a voiceproximity sensor and can also be combined with an amplification sensorfor the sound wave, as well as at least one directional sensor. In thismanner, an individual speaking a command (such as “turn lights on” or“turn lights off”) can direct the command to a specific endpoint 22, 24,26, 28 within a room or a specific room as they enter or leave in orderto distinguish from an endpoint in the adjacent room.

Bridge Functions:

The present invention includes perception of voice commands, coding,transmitting to remote voice web-service 84 (Amazon Alexa, GoogleAssistant, etc.) using protected HTTP connection. This includes:receiving, uncoding, unpacking and playing of sound/voice response fromremote voice web-service. There is also receiving of REST-requests fromown web-service (NMA) and control of devices with the help of GPIO's 52pins or using wireless interfaces. See FIG. 3 to reference the audiodata driver 62, communicating and transmitting to data conversion 64which is in communication with the network layer 70 and business logiclayer 66. The business logic layer 66 communicates with the GPIO driver52 and other device drivers 68. The business logic layer alsocommunicates with the network layer 70 which is in communication withthe network 72.

NMA Functions:

Referring to FIG. 4, there is shown the bridge system diagram 80. Thisbridge system includes a multitude of ESP based bridges 90, 92, 94connected and communicating with a Wi-Fi router 88 in connection to theinternet 86. Communication with an NMA 82 and a speech recognitionservices 84 to and from the internet 86 is also provided.

NMA 82 is a web service that contains event handlers for voice webservices. It handles requests from a remote voice web service (AmazonAlexa, Google Assistant, etc) 84. It sends REST bridge requestsaccording to its own business logic, which is based on processing eventsfrom a remote voice web service.

Functions of the Remote Voice Web Service.

This service has the functionality to recognize voice information, theformation of a voice response based on intellectual processing of inputdata (contains intellectual voice chat) and also contains easilyconfigurable voice command handlers (e.g. Alexa Skills) and NMA webservice management.

Working Flow:

After power supply to the bridge, the device enters the standby mode ofinitialization, which is displayed by the indicator. The device isinitialized by pressing the “mic” button or by pre programmed wake-upword. In the initial initialization mode, the bridge raises the accesspoint with the SSID (brige_xxxxx). This is necessary to configure thebasic parameters such as WIFI AP and voice web service account 84. Setupis performed using a mobile IOS/Android application. The user installsthe mobile application. The mobile device must be connected to the WIFIAP bridge. After successful setting, the bridge disables the accesspoint. To reset the settings, you must hold the “reset” button.

The configured bridge connects to the NMA 82 and also has a connectionto the remote voice web service 84. After successfully connecting to theNMA 82, the bridge is waiting for the wake-up voice command word. Theuser has the ability to customize the wake-up word voice command using amobile application. User information will be stored in the bridge ROM inencrypted form. The key for encryption is located in a secure section ofthe flash. These states are accompanied by light/sound indication.

The user initiates voice control of bridge by the wake-up word. Afterprocessing of wake-up word, the bridge goes into the mode oftransmitting voice information to the voice service. A voicecommunication session has a specified timeout upon completion of whichcommands are not transmitted to the voice service. For subsequentsessions, you must repeat the pronunciation of wake-up word.Initialization of communication sessions is accompanied by a light/soundindication. The voice service receives voice information from thebridge, processes the request, sends an audio response to the bridge,and, if necessary, transmits the necessary request to the NMA. NMA inturn controls the bridge. (See FIG. 4)

FIG. 5 illustrates a system 500 of a computer or device which includes amicroprocessor 520 and a memory 540 which are coupled to a processor bus560 which is coupled to a peripheral bus 600 by circuitry 580. The bus600 is communicatively coupled to a disk 620. It should be understoodthat any number of additional peripheral devices are communicativelycoupled to the peripheral bus 600 in embodiments of the invention.Further, the processor bus 560, the circuitry 580 and the peripheral bus600 compose a bus system for computing system 500 in various embodimentsof the invention. The microprocessor 520 starts disk access commands toaccess the disk 620. Commands are passed through the processor bus 560via the circuitry 580 to the peripheral bus 600 which initiates the diskaccess commands to the disk 620. In various embodiments of theinvention, the present system intercepts the disk access commands whichare to be passed to the hard disk.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” or“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The “computer readable storage medium” may be any tangible medium (butnot a signal medium—which is defined below) that can contain or store aprogram. The terms “machine readable medium,” “computer-readablemedium,” or “computer readable storage medium” are all non-transitory intheir nature and definition. Non-transitory computer readable mediacomprise all computer-readable media except for a transitory,propagating signal.

The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor. A“computer readable signal medium” may be any computer readable mediumthat is not a computer readable storage medium and that can communicate,propagate, or transport a program.

What is claimed is:
 1. A voice orchestrated infrastructure systemcomprising, a hub in communication with at least one endpoint devicelocated in a room or area, said at least one endpoint device incommunication with said hub and at least one endpoint device in a secondroom or area through said hub, said hub including a set ofnon-transitory commands which when executed with a central processorsaid at least one endpoint device activated and controlled by voicecommands which are independent of service provider type; said hub havinga non-transitory computer-readable storage medium which storescomputer-executable instructions that, when executed by a processor,cause the processor to perform operations for determining the voicecommand which is communicated to and from said at least one endpointdevice.